1. Field of the Invention
The present invention relates to a method and apparatus for obtaining information that expresses image deformation between frame images of a moving image caused by shake of a camera or the like in photographing moving images.
2. Related Background Art
Conventionally, various methods have been proposed concerning a technique for correcting an image blur due to shake of a camera such as a hand shake in an image pickup apparatus such as a video camera. In addition, there are various methods concerning an image movement detection technique that constitutes an image blur correction technique. As one of the techniques, there is a technique for detecting plural movement vectors between frame images forming a moving image and calculating a movement vector representing a whole screen from those movement vectors to thereby detect movement of an image. However, image movement detecting methods and apparatuses, which are generally used for correction of an image blur, detect only movement of an image in horizontal and vertical directions. Thus, there is a problem in that, when a complicated image blur due to factors including rotation of an image pickup apparatus occurs, it is impossible to accurately detect the blur and the blur remains in a photographed image.
A movement vector caused by shake of a camera is a sum of movement components of magnification and reduction, rotation, and horizontal and vertical movements of an image. Thus, in order to accurately detect the horizontal movement and the vertical movement, influences of other shakes such as the magnification and reduction and the rotation have to be removed. Therefore, there is a demand for development of an image movement detection technique that is capable of detecting movement such as rotation in addition to movements in horizontal and vertical directions or detecting components of individual shakes accurately.
Deformation (geometrical deformation) of an image such as magnification and reduction, rotation, horizontal movement, and vertical movement is caused by camera shake between frame images. Affine transformation is one of models expressing the deformation. The image deformation caused by the camera shake can be expressed in a form of a matrix as indicated by the following expression.
                              [                                                                      x                  ′                                                                                                      y                  ′                                                              ]                =                                            [                                                                    a                                                        b                                                                                                              -                      b                                                                            a                                                              ]                        ⁡                          [                                                                    x                                                                                        y                                                              ]                                +                      [                                                            c                                                                              d                                                      ]                                              (        1        )            
A movement vector caused by the camera shake can be represented by the following expression.
                              [                                                                      v                  x                                                                                                      v                  y                                                              ]                =                                                            [                                                                                                    a                        ′                                                                                                            b                        ′                                                                                                                                                -                                                  b                          ′                                                                                                                                    a                        ′                                                                                            ]                            ⁡                              [                                                                            x                                                                                                  y                                                                      ]                                      +                          [                                                                                          c                      ′                                                                                                                                  d                      ′                                                                                  ]                                =                                                    [                                                                                                    a                        -                        1                                                                                    b                                                                                                                          -                        b                                                                                                            a                        -                        1                                                                                            ]                            ⁡                              [                                                                            x                                                                                                  y                                                                      ]                                      +                          [                                                                    c                                                                                        d                                                              ]                                                          (        2        )            where, (x, y) represents arbitrary coordinates on the image and (x′, y′) represents coordinates to which the coordinates (x, y) are moved by the camera shake, (vx, vy) represents an x component and a y component of a movement vector in the coordinates (x, y), a, b, c, and d are coordinate conversion parameters representing image deformation caused by the camera shake and a′, b′, c′, and d′ are movement parameters representing movement of the image caused by the camera shake.
When this affine transformation is used, it is possible to express the deformation and the movement vector of the image in arbitrary coordinates on the image, the deformation being caused by the cameral shake, with four parameters. In other words, it is possible to detect deformation or movement of the entire image by obtaining a, b, c, and d when attention is paid to the image deformation and by using a′, b′, c′, and d′ when attention is paid to the movement of the image.
Techniques for detecting movement of an image using the model described above are disclosed in Japanese Patent No. 2586686, Japanese Patent No. 2506500, Japanese Patent Application Laid-Open Nos. H9-212649, H10-233958, and the like.
In the method disclosed in Japanese Patent No. 2586686, the least square error estimation is applied to a movement vector, which is detected for each pixel or each small area, and a model of movement caused by a camera shake to detect a movement parameter expressing movement of an entire image. In addition, an error in vector detection is removed by using a filter to improve detection accuracy.
However, in the method disclosed in Japanese Patent No. 2586686, when there is movement different from a camera shake such as movement of a subject in a relatively large area, it is impossible to completely remove the movement with the filter. Thus, the accuracy is not enough from the viewpoint of detection of movement caused by the camera shake.
In the method disclosed in Japanese Patent No. 2506500, using conversion coefficients calculated by using plural movement vectors in a position of an image, an amount of movement in other positions of the image is predicted. Then, movement of the entire image is estimated using only movement vectors in an area where the number of areas, where a difference between the predicted value and an actually detected movement vector is equal to or lower than a threshold value, takes the maximum value.
In the method disclosed in Japanese Patent No. 2506500, an amount of calculation is large because it is repeatedly checked whether movements are identical on the basis of a difference between a movement vector calculated by using the conversion coefficients and the detected vector.
In Japanese Patent Application Laid-Open No. H9-212649, a movement parameter is calculated using plural movement vectors out of the movement vectors detected locally, and the calculation is repeated to calculate plural movement parameters. Then, a movement parameter value having a high frequency of appearance among the movement parameters is estimated as a movement parameter value representing an entire image.
In the method disclosed in Japanese Patent Application Laid-Open No. H9-212649, the movement parameter having a high frequency of appearance is set as a parameter for the entire image. However, it is likely that plural peaks exist or a wrong parameter is determined as a parameter value for the entire image when no clear peak is present.
In the method disclosed in Japanese Patent Application Laid-Open No. H10-233958, a distribution in a characteristic space of a movement vector is taken into account to calculate a parameter value for basic camera operation. In the method disclosed in Japanese Patent Application Laid-Open No. H10-233958, it is possible to calculate a parameter value for each camera shake. However, complicated processing such as clustering is required.
For image blur correction in a photographing apparatus such as a video camera, it is necessary to detect movement of an image due to a camera shake at high speed and accurately under various conditions. Therefore, it is necessary to develop an image movement detecting method and an image movement detecting apparatus that can solve the problems of the techniques described above and detect movement of an image due to a camera shake at high speed and accurately without being affected by movement of a subject and an error in movement detection and without being affected by photographing conditions. In addition, when correction of an image blur is taken into account, a coordinate conversion parameter expressing image deformation is more advantageous as a parameter to be detected than a movement parameter expressing a movement vector in the image.
Since the degree of movement due to magnification and reduction or rotation is proportional to a distance from an image center, minute movement due to magnification and reduction or rotation often appears only in a peripheral part of an image. However, in the methods described above, since movement vectors on the image used for calculating parameters expressing movement of the entire image are treated uniformly, it may be impossible to detect minute movement that appears only in the peripheral part of the image. Although an error in the movement detection is very small, when a parameter between frame images apart from each other is calculated by accumulating parameters among adjacent frame images, the error is accumulated to cause inconvenience.
Moreover, all the methods described above do not take into account a moving image having distortion photographed by a fish-eye lens or the like.